Modular Approach to Error Analysis and Evaluation for Multilingual Question Answering

The JAVELIN system is a modular, extensible and language-independent architecture for building question-answering systems [3]. We have been working to extend the original English version of JAVELIN for cross-language question answering in Chinese and Japanese. Recently, we participated in the NTCIR5 CLQA11 evaluation. Out of 13 groups participating in the CLQA1 task, we are the only group to submit formal runs for both the English-to-Chinese (EC) and the English-to-Japanese (EC) subtasks. After analyzing the observed performance of each module on the evaluation data, we created gold standard data (perfect input) for each module in order to determine upper bounds on module performance. This modular approach allows us to compare the performance of the two systems (EC and EJ) on a per-module basis.